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Abstract-  A  Technical  Metrics  Workshop  was  held  at  NRL  Stennis  Space  Center,  MS,  by  the  NRL  Acoustics  Division  April  29  -  May 
1,  2008.  The  primary  goal  of  the  workshop  was  to  identify  the  state-of-the-art  technical  and  scientific  metrics  for  the  acoustic  and 
oceanographic  research  and  development  communities.  A  second  goal  was  to  relate  scientific  metrics  to  engineering  and  performance 
metrics,  and  outline  the  inclusion  of  uncertainties  in  various  parameters  using  metric  variability. 

Participation  in  the  workshop  was  significant.  State-of-the-art  technical  applications,  as  well  as  brief  insights  into  future  metrics, 
requirements  for  metrics,  standards  or  reference  level  metrics  were  presented.  This  paper  briefly  summarizes  and  presents  the 
conclusions  of  the  workshop.  This  work  is  a  useful  reference  for  building  consistent  technical  metrics  within  the  ocean  and  acoustic 
research  and  development  community,  as  well  as  providing  common  guidance  for  measuring  exit  and  milestone  criteria,  and  for 
quantifying  the  contributions  of  interdisciplinary  projects. 

I.  Introduction 

Scientific  /  technical  metrics  are  widely  used  by  various  communities  at  multiple  levels  from  basic  scientific  analysis  to 
decision  making.  The  Navy  and  other  communities  have  specific  requirements  that  could  benefit  directly  from  such  metrics.  The 
overarching  goal  of  this  workshop,  held  at  NRL  Stennis  April  28-May  1,  2008,  was  to  have  an  interdisciplinary  discussion  on 
how  these  ongoing  efforts  could  meet  higher  level  needs  (e.g.  decision  makers).  Under  this  approach,  more  specific  goals  of  the 
workshop  were  then  to  describe  and  identify  technical  and  scientific  metrics,  build  on  current  methods  and  discuss  how  these 
tools  could  be  adapted  /  applied  to  operational  and  management  metrics  in  the  future. 

II.  Participants 

The  NRL  Technical  Metrics  workshop  committee  2008  was  lead  by  Josette  Fabre,  Naval  Research  Laboratory,  Stennis  Space 
Center  (NRL  SSC)  Code  7180  (acoustics).  Other  members  of  the  committee  included  (in  alphabetical  order)  Emanuel  Coelho, 
NRL  SSC  Code  7320  (oceanography)  /  University  of  Southern  Mississippi  (USM),  James  Dykes,  NRL  SSC  Code  7320 
(oceanography),  Pat  Gallacher,  NRL  SSC  Code  7330  (oceanography),  Roger  Gauss,  NRL  Code  7140  (acoustics),  Dr.  Joe 
Metzger,  NRL  SSC  Code  7320  (oceanography)  and  Dr.  Tom  Murphree,  Naval  Postgraduate  School  (NPS)  (meteorology/ 
oceanography/  climate/metrics).  Contributors  to  the  workshop  and  report  included  Dr.  Merrill  Stevens,  Commander,  Naval 
Meteorology  and  Oceanography  Command  (CNMOC),  Dennis  Krynen,  Naval  Oceanographic  Office  (NAVO),  Keith  Atkinson 
(NAVO),  Steve  Woll  (Weatherflow,  Inc.),  Dr.  Bill  Stevens  (Applied  Operations  Research),  Dr.  Tom  Murphree  (NPS),  Bruce 
Ford  (Clear  Science,  Inc.),  Barbara  Brown,  National  Center  for  Atmospheric  Research  (NCAR),  Dr.  Gregg  Jacobs  (NRL  SSC), 
Joe  Metzger  (NRL  SSC),  Dr.  Tim  Duda  Woods  Hole  Institute  (WHOI),  Dr.  Peter  Mignerey  (NRL),  Dr.  Roger  Oba  (NRL),  Dr. 
Orest  Diachok,  Johns  Hopkins  University  /  Applied  Physics  Laboratory  (JHU/APL),  Dr.  Roger  Gauss  (NRL),  John  Perkins 
(NRL),  Dr.  James  Fulford  (NRL),  J.  Paquin  Fabre  (NRL),  Steven  Dennis  (NRL),  Dr.  Robert  Miyamoto  Applied  Physics 
Laboratory/  University  of  Washington  (APL/UW),  Dr.  Warren  Wood  (NRL),  Dr.  Pat  Cross,  Ocean  Acoustical  Services  and 
Instrumentation  Systems  (OASIS),  Emanuel  Coelho  (USM  /  NRL),  Dr.  Steve  Finette  (NRL),  Kevin  Heaney  (OASIS),  Dr.  Paul 
Elmore  (NRL),  Dr.  Roger  Gauss  (NRL),  Dr.  Ray  Mahdon  (United  Kingdom  MetOffice),  Dr.  Frank  Bub  (NAVO),  James  Dykes 
(NRL),  Dr.  Chad  Steed  (NRL),  and  Bruce  Northridge  (CNMOC). 

III.  The  Workshop 

The  workshop  was  organized  as  follows.  During  the  introduction  or  motivation,  the  shortfalls  of  operational  requirements  that 
could  benefit  from  application  of  some  type  of  metrics  analyses  were  presented.  Next,  the  state-of-the-art  metrics  were  presented, 
emphasizing  how  efforts  could  be  steered  towards  the  relevant  problems.  The  state-of-the-art  talks  were  categorized  into  the 
subject  areas  of  Meteorology  and  Oceanography  (METOC),  acoustics,  bottom  and  uncertainty.  Finally,  technical  metrics  were 
related  to  higher  level  decision  making,  addressing  the  questions  on  how  to  assimilate  metrics  of  different  types,  along  with  other 
approaches  to  address  the  shortfalls  identified  at  the  beginning  of  the  workshop.  These  efforts  identify  our  current  technical  / 
scientific  metrics  state,  identify  shortfalls  and  research  directions  and  help  design  an  end-to-end  roadmap  for  the  future. 

The  workshop  emphasized  how  direct  observations  and  model  estimates  can  be  combined  to  provide  better  operational 
guidance  and  assist  in  solving  specific  end-user  requirements.  This  will  allow  enhanced  quantitative  confidence  for  the  end-user, 
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and  therefore,  provide  not  just  an  improved  “final  answer”  but  also  identify  scenario-dependent  performance  drivers  and  provide 
estimates  of  uncertainty  that  can  more  reliably  and  robustly  inform  operational  decisions. 

IV.  State-of-the-Art  Metrics 

The  focus  of  this  workshop  was  on  acoustic  performance  prediction,  therefore  the  models  and  data  used  to  drive  the  acoustic 
models  as  well  as  acoustic  models  and  analysis  techniques,  were  presented  and  discussed.  The  ocean  and  acoustic  models  were 
introduced  by  subject  matter  experts  from  NAVO.  They  stressed  the  need  for  timely  products  and  a  way  to  communicate  product 
confidence  to  end  users.  The  ocean  model  output  is  highly  dimensional  in  space  and  time;  because  of  the  computation  times 
required  to  compute  acoustic  performance  predictions,  derived  parameters  and  change  metrics  are  required  to  determine  how  to 
choose  appropriate  input  scenarios  for  the  acoustic  performance  predictions.  Verification  and  Validation  (V&V)  metrics  are  also 
required  to  compare  predictions  to  data  and  indicate  acceptable  variations. 

Due  to  the  many  aspects  of  the  acoustic  dependence  on  the  environment  (surface,  water,  bottom),  there  are  multiple  sources  of 
inaccuracy  or  uncertainty.  Such  errors  need  to  be  propagated  through  to  the  acoustic  answer  and  communicated  to  the  end  users. 

Significant  work  has  been  done  in  end-user  metrics  [1].  There  are  many  levels  of  metrics  that  can  be  provided  to  the  end-users, 
for  example,  quantification  of  the  impact  of  various  quantities  (e.g.  environmental)  and  how  well  they  are  represented,  on  the  end 
products  (e.g.  performance  estimates).  Metrics  should  always  be  developed  with  the  end  goal  in  mind. 

Metrics  were  presented  from  a  wide  variety  of  disciplines  for  multiple  purposes  including,  but  not  limited  to,  various  direct 
comparisons  of  measurements  to  models,  spatial  feature  analysis  of  weather  models  and  its  potential  application  to  ocean  models, 
sampling  metrics  for  determination  of  sensor  placement  and  resolution  requirements,  metrics  for  determining  the  impact  of  a 
measurement  on  a  predicted  performance  and  its  accuracy,  coherence  metrics  in  signal  processing,  metrics  showing  the  impact  of 
mis-represented  data  in  models,  specialized  curve  comparison,  quantifying  uncertainty  and  acoustic  performance  metrics. 

V.  Results 

It  was  shown  that  mature  metrics  exist  in  each  of  the  categories  of  metrics  that  are  appropriate  to  this  community:  scientific  / 
technical,  performance  and  operational.  The  community  has  impressive  metrics  capabilities  within  the  science  and  engineering 
domains.  Similarly  well-defined  operations  metrics  have  been  developed  within  the  Navy  operations  research  community.  What 
is  missing  is  a  general-purpose  approach  for  tracing  scientific  improvements  (e.g.  a  better  temperature  and  salinity  forecast)  to 
engineering  impacts  (e.g.  resulting  improved  ability  to  estimate  sonar  performance)  to  warfighting  impacts  (e.g.  resource 
allocation  and  time  to  operate).  This  methodology  must  provide  the  means  to  also  trace  uncertainties  and  errors  from  METOC 
data  collection,  assimilation,  and  modeling  to  end-user  operational  effectiveness:  —  correcting  this  shortfall  is  a  primary  long¬ 
term  objective  of  the  NRL  Technical  Metrics  Committee  (NTMC). 

It  was  determined  that  the  metric  transmission  loss  (TL)  difference  and/or  figure  of  merit  (FOM)  difference  with  enough 
information  for  uncertainty  or  sensitivity  will  provide  a  common  scientific  /  technical  assessment  that  can  be  computed  at  the 
output  of  each  process  and  can  then  be  easily  translated  to  performance  quantities.  The  FOM  is  used  to  translate  the  TF  to  a  more 
system  specific  quantity  of  detectable  signal,  or  signal  excess. 

The  Navy  acoustics  S&T  community  is  comfortable  with  TF,  but  much  less  so  with  FOM  because  of  the  inclusion  of 
equipment  capabilities,  operator  proficiency,  and  other  factors  which  are  hard  to  quantify  in  a  scientifically  rigorous  fashion. 
Nevertheless,  TF  is  of  limited  value  operationally  without  an  estimate  of  a  FOM  value  or  distribution  of  possible  FOM  values. 
This  represents  a  disconnect  between  the  operational  and  the  Navy  acoustics  communities. 

Figure  1  shows  the  overall  acoustic  performance  prediction  process  generated  at  the  workshop.  It  begins  with  the  descriptions 
of  the  environment  in  terms  of  the  measured  and  databased  quantities.  Note  that  each  process  has  an  associated  uncertainty  that 
must  be  considered  in  some  fashion.  These  input  quantities  are  fed  to  the  acoustic  model  and  sensitivities  to  environmental  inputs 
and  metrics  are  generated.  Sonar  performance  predictions  are  made  using  the  acoustic  model  outputs  or  derived  metrics.  The 
performance  predictions  are  passed  to  decision  makers  or  operators.  Assessments  of  the  validity  of  the  performance  model  can  be 
fed  back  (based  on  actual  performance  or  other  knowledge)  to  the  acoustic  model  for  modification  of  inputs  or  modifications  to 
the  model  itself.  Decisions  based  on  these  assessments  are  passed  to  the  operators.  After  an  exercise  is  complete,  reconstruction 
and  analysis  (R&A)  lessons  learned  can  be  fed  back  to  all  aspects  of  the  process. 
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Figure  1.  Schematic  of  how  metrics  relate  to  various  stages  of  operations. 


VII.  An  Example 

An  example  of  applying  a  technical  or  scientific  metric  is  given  here  to  demonstrate  the  process  of  tracking  metrics  from  the 
level  of  measurements  or  data,  all  the  way  to  their  end  use.  A  sensor  has  been  developed  that  will  provide  high  spatial  resolution 
temperature  and  salinity  data  for  a  relatively  small  area  and  we  would  like  to  assess  the  impact  of  such  data  on  a  real-world 
application  such  as  our  ability  to  predict  the  performance  of  an  acoustic  sonar  detecting  the  bottom  or  objects  on  the  bottom.  An 
ocean  model  is  run  with  and  without  assimilating  the  high  resolution  oceanographic  data,  and  the  sound  velocity  is  predicted  for 
an  area  of  interest.  This  provides  assessments  of  the  environment  that  represent  both  the  new  capability  and  the  original 
capability.  These  assessments  are  input  to  an  acoustic  propagation  model  and  transmission  loss,  with  uncertainty,  is  predicted  for 
both  cases  and  compared.  Figure  2  shows  an  example  comparison  between  two  TL  curves,  one  using  a  low  resolution  ocean 
description  to  simulate  an  existing  capability  (blue)  and  the  other  a  high  resolution  ocean  description  to  simulate  an  improved 
measurement  capability.  Appropriate  figures  of  merit  (with  associated  uncertainty)  are  applied  and  the  detectable  signal  is 
assessed.  In  Figure  2,  this  FOM  is  indicated  by  the  red  line.  Above  the  line,  objects  can  be  detected,  and  below  the  line  they  can’t. 
In  this  example,  the  original  prediction  was  over-optimistic,  indicated  that  objects  would  be  detectable  to  a  range  of 
approximately  1 600m,  whereas  the  prediction  using  the  higher  resolution  ocean,  indicates  detections  only  to  about  1 1 00m.  In 
general,  single  values  with  error  bars  or  probability  distribution  functions  are  then  input  to  the  performance  prediction  model  to 
compute  values  such  as  probability  of  detection  or  sensor  coverage,  and  these  values  are  used  to  make  decisions.  The  resulting 
differences  in  the  performance  predictions  and  the  decisions  made  can  be  assessed  and  the  impact  of  the  higher  resolution 
oceanographic  data  can  be  quantified.  Here,  a  decision  regarding  the  sensor  range  coverage  would  have  been  in  error  by 
approximately  500m.  This  is  a  simplistic  case,  but  can  be  expanded  with  uncertainty  information  and  other  metrics,  for  example, 
area  coverage  for  the  entire  area,  or  probability  of  detection  over  the  area  and  the  differences  can  be  communicated  to  decision 
makers  in  terms  that  they  are  familiar  with.  A  conclusion  using  this  example  might  be  that  an  improved  performance  prediction 
provided  TL  that  was  more  accurate  by  up  to  10  dB  and  resulted  in  an  improved  range  estimate  of  approximately  500m.  These 
metrics  can  be  take  further  to  assess  the  problem  in  terms  of  sensor  placement  or  allocation. 
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Figure  2.  TL  predicted  using  original  profile  (blue)  compared  to  TL  predicted  using  higher  resolution  oceanographic  data  (green). 

The  red  line  is  a  generic  FOM  that  indicates  what  part  of  the  predicted  range  would  be  "visible"  by  the  sensor. 

VIII.  The  Way  Ahead 

The  METOC  R&D  community  is  now  thinking  along  the  lines  of  how  their  metrics  need  to  be  presented  to  the  operational  and 
decisional  communities.  A  preliminary  way  ahead  follows. 

The  decision  making  community  would  like  to  quantify  the  value-added  of  METOC  assimilation,  modeling,  database,  or  TDA 
developments.  The  operational  community  would  like  this  information  in  addition  to  how  to  obtain  and  employ  new  capabilities. 
Another  important  aspect  of  this  is  that  for  a  new  capability  to  be  fielded,  the  resulting  capability  improvements  need  to  be 
substantial  enough  to  warrant  the  cost  of  transitioning  to  the  new  system. 

The  scientific  community  must  be  able  to  express  their  impacts  in  changes  in  transmission  loss  (TL)  or  figure  of  merit  (FOM). 
For  many,  that  requires  running  an  acoustic  model.  Different  models  apply  to  different  applications  and  each  scientist  should 
understand  which  models  apply  to  their  particular  research  application. 

Because  of  this  TDAs  have  been  developed  that  try  to  deal  with  the  FOM  issue  directly.  Moving  from  TL  to  the  other  terms  in 
the  sonar  equation  could  hence  be  part  of  a  new  focus  on  metrics. 

Next,  a  general-purpose  approach  for  tracing  scientific  improvements  that  have  been  expressed  in  terms  of  TL  or  FOM,  to 
engineering  impacts  to  warfighting  is  being  developed  and  will  be  made  available  to  the  community.  This  methodology  must  be 
capable  of  tracing  uncertainties  and  errors  from  METOC  data  collection,  assimilation,  and  modeling  to  end  user  operational 
effectiveness.  It  must  be  as  simple  as  possible  so  as  to  be  relevant  to  multiple  applications,  with  the  knowledge  that  further 
analysis  may  be  required.  This  effort  must  be  coordinated  with  the  existing  capabilities  on  both  ends  (e.g.  N81/N84/CNMOC  and 
NAVO/NRL/R&D  community)  so  as  to  provide  consistent  and  agreed  upon  results.  Some  capabilities  do  currently  exist,  but  are 
likely  not  in  a  format  that  can  be  easily  used  by  the  S&T  community. 

Metrics  such  as  sonic  layer  depth  and  cutoff  frequency  considered  with  the  sensor  configuration  is  an  example  of  using 
environmental  factors  to  compute  a  measure  that  is  operationally  meaningful.  There  are  other  approaches  as  well.  The 
committee  should  be  able  to  define  a  range  of  approaches,  building  on  work  done  in  the  studies  briefed  at  the  workshop  and 
elsewhere  and  put  this  together  as  a  "code  of  best  practice"  for  estimating  METOC  impacts  on  warfighting  effectiveness.  DoD 
and  NATO  have  developed  something  similar  called  the  "C4I  Analysis  Code  of  Best  Practice"  [2]. 

The  next  step  in  this  metrics  process  will  then  be  to  research,  identify  and  propose  an  approach  or  approaches  for  development 
of  the  aforementioned  methodology.  This  approach  will  be  different  for  various  systems,  but  the  initial  focus  will  be  on  current 
systems  discussed  during  the  workshop.  An  example  approach  would  be  to  develop  a  generic  scenario  for  which  environmental 


acoustic  products  and  their  variations  can  be  applied  to  quantify  the  operational  impacts  of  the  product  in  terms  that  can  be 
communicated  to  decision  makers.  Stevens  et  al.  (2008)  provide  examples  that  can  be  followed  for  this  type  of  approach. 

It  is  expected  that  the  scientists  that  participated  in  this  workshop  will  continue  to  develop  and  improve  their  metrics  for  more 
easy  translation  to  higher  level  metrics.  It  is  well  understood  that  there  is  no  “silver  bullet”  metric,  however,  steps  can  be  and  are 
being  made  to  bring  the  two  communities  closer  together. 

Future  directions  in  this  area  should:  1 .  present  new  technical  metrics  and  progress  on  existing  technical  and  related  metrics 
since  the  2008  workshop;  2.  develop  and  refine  the  general  procedure  for  deriving  operational  metrics  from  technical  metrics; 
document  the  issues  involved;  and  potentially  to  begin  applying  the  procedure  to  a  test  case;  and  3.  get  feedback  from  the  various 
entities  involved  on  the  technical  metrics  way  ahead. 
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